{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f923a7f9-5505-4fbe-be4a-8870d9b2d7cf",
   "metadata": {},
   "source": [
    "# MultiOn Demo\n",
    "\n",
    "<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/llama-index-integrations/tools/llama-index-tools-multion/examples/multion.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>\n",
    "\n",
    "This notebook walks through an example of using LlamaIndex with MultiOn to browse the web on the users behalf.\n",
    "\n",
    "First, we import the FunctionAgent that will control the Multion session:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "daea8a25-468e-470c-99c9-ee6429bc522f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up OpenAI\n",
    "import os\n",
    "\n",
    "os.environ[\"OPENAI_API_KEY\"] = \"sk-your-key\"\n",
    "\n",
    "from llama_index.core.agent.workflow import FunctionAgent\n",
    "from llama_index.llms.openai import OpenAI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4e2a467-bb2b-4642-993d-80b1fc852add",
   "metadata": {},
   "source": [
    "We then import the MultiOn tool and initialize our agent with the tool."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77d5b1ef-876f-4b90-94e4-dfde91e77fed",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up MultiOn tool\n",
    "from llama_index.tools.multion import MultionToolSpec\n",
    "\n",
    "multion_tool = MultionToolSpec(api_key=\"your-multion-key\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1d1d2f87-5a1f-4722-9f4f-ad0e072b190d",
   "metadata": {},
   "source": [
    "To support the MultiOn browsing session, we will also give our LlamaIndex agent a tool to search and summarize a users gmail inbox. We set up that tool below. For more information on the gmail tool, see the [Gmail notebook here](https://github.com/emptycrown/llama-hub/blob/main/llama_hub/tools/notebooks/gmail.ipynb).\n",
    "\n",
    "We will use this tool later on to allow the agent to gain more context around our emails"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f74cab58-1b97-416e-bb7a-ee7eaa14555a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import and initialize our tool spec\n",
    "from llama_index.tools.google import GmailToolSpec\n",
    "from llama_index.core.tools.ondemand_loader_tool import OnDemandLoaderTool\n",
    "\n",
    "# Initialize the Gmail tool to search our inbox\n",
    "gmail_tool = GmailToolSpec()\n",
    "\n",
    "# Wrap the tool so we don't overflow the main Agent's context window\n",
    "gmail_loader_tool = OnDemandLoaderTool.from_tool(\n",
    "    gmail_tool.to_tool_list()[1],\n",
    "    name=\"gmail_search\",\n",
    "    description=\"\"\"\n",
    "        This tool allows you to search the users gmail inbox and give directions for how to summarize or process the emails\n",
    "\n",
    "        You must always provide a query to filter the emails, as well as a query_str to process the retrieved emails.\n",
    "        All parameters are required\n",
    "        \n",
    "        If you need to reply to an email, ask this tool to build the reply directly\n",
    "        Examples:\n",
    "            query='from:adam subject:dinner', max_results=5, query_str='Where are adams favourite places to eat'\n",
    "            query='dentist appointment', max_results=1, query_str='When is the next dentist appointment'\n",
    "            query='to:jerry', max_results=1, query_str='summarize and then create a response email to jerrys latest email'\n",
    "            query='is:inbox', max_results=5, query_str='Summarize these emails'\n",
    "        \"\"\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee9b63f5-4f99-4708-b1ab-67d64fdb0cee",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize our Agent with the MultiOn and Gmail loader tool\n",
    "agent = FunctionAgent(\n",
    "    tools=[*multion_tool.to_tool_list(), gmail_loader_tool],\n",
    "    system_prompt=\"\"\"\n",
    "    You are an AI agent that assists the user in crafting email responses based on previous conversations.\n",
    "    \n",
    "    The gmail_search tool connects directly to an API to search and retrieve emails, and answer questions based on the content.\n",
    "    The browse tool allows you to control a web browser with natural language to complete arbitrary actions on the web.\n",
    "    \n",
    "    Use these two tools together to gain context on past emails and respond to conversations for the user.\n",
    "    \"\"\",\n",
    "    llm=OpenAI(model=\"gpt-4.1\"),\n",
    ")\n",
    "\n",
    "# Context to store chat history\n",
    "from llama_index.core.workflow import Context\n",
    "\n",
    "ctx = Context(agent)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d16f81c-ce52-4405-9bb7-e5950123df8b",
   "metadata": {},
   "source": [
    "Our agent is now set up and ready to browse the web!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a159dfe7-a3dc-4774-870c-23687611ffb1",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(await agent.run(\"browse to the latest email from Julian and open the email\", ctx=ctx))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cdf45a41-6009-46e5-a809-8d818c3b4d0f",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\n",
    "    await agent.run(\n",
    "        \"Summarize the email chain with julian and create a response to the last email\"\n",
    "        \" that confirms all the details\",\n",
    "        ctx=ctx,\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07fe11f9-da88-4eed-baa8-020a1e795cbb",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\n",
    "    await agent.run(\n",
    "        \"pass the entire generated email to the browser and have it send the email as a\"\n",
    "        \" reply to the chain\",\n",
    "        ctx=ctx,\n",
    "    )\n",
    ")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
